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Efficient predictive image parameter estimation 



The invention relates to a method for recursively estimating local vectors from 
at least one picture taken from an image sequence, comprising the steps of generating a first 
set of candidate vectors under at least partial use of recursion, selecting candidate vectors 
from the first set of candidate vectors according to a first criterion to form a smaller second 
5 set of candidate vectors, evaluating the candidate vectors of the second set of candidate 
vectors for a group of pixels based on a second criterion, determining the best vectors from 
the second set of candidate vectors according to said second criterion and assigning said 
determined best vectors to a group of pixels that is related to the group of pixels the candidate 
vectors of the second set of candidate vectors were evaluated for. The invention further 
10 relates to a device for recursively estimating local vectors from at least one picture taken 
from an image sequence, and to a computer program product comprising software code 
portions for recursively estimating local vectors from at least one picture taken from an 
image sequence. 

15 

Estimation of local vectors from image data is required for a broad range of 
image processing applications, such as coding/compression, noise reduction* object tracking 
and scan rate conversion. In a video coding framework such as MPEG or H.261, local vectors 
are represented by motion vectors that determine motion (or object displacement) from one 

20 image to another. Estimation of motion vectors can for instance be used for motion- 
compensated predictive coding. Since one picture in an image is normally very similar to a 
displaced copy of its predecessors, encoding estimated motion vector data together with 
information on the difference between the actual image and its prediction either in the pixel- 
or DCT-domain allows to vastly reduce the temporal redundancy in the coded signal. 

25 Further examples for the estimation of local vectors comprise methods to 



segment an image in areas with similar spatial characteristics (object segmentation), where 
the local vectors the* represent a quantitative measure for the spatial characteristics, and 
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to estimate local vectors ftom image data usually 
, where an image is decomposed in 

' (object segmentation), so that the subsequent description equally 
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computational complexity, To assure accurate motion vector estimation, a large set of 
candidate motion vectors has to be chosen, which leads to an increased computational 
complexity. 



It is thus the object of the invention to provide a recursive method for accurate 
estimation of local vectors with reduced complexity and fast convergence. 

To solve the object of the invention it is proposed that the method for 
recursively estimating local vectors from at least one picture taken from an image sequence 
comprises the steps of generating a first set of candidate vectors under at least partial use of 
recursion, selecting candidate vectors fean the first set of candidate vectors according to a 
first criterion to form a smaller second set of candidate vectors, evaluating the candidate 
vectors of the second set of candidate vectors for a group of pixels based on a second 
criterion, deteimiomg me best vectors from the second set of candidate vectors according to 
15 said second criterion and assigning said determined best vectors to a group of pixels that is 
related to the group of pixels the candidate vectors of the second set of candidate vectors 
were evaluated for. 

By reducing the size of the first set of candidate vectors according to the first 
criterion, the similarity measure according to the second criterion has to be evaluated for less 

20 candidate vectors, so that the computational complexity can be vastly decreased as compared 
to state-of-the-art estimators, where the similarity measure has to be evaluated for all 
candidate vectors of the first set of candidate vectors. The first criterion has a low 
computational complexity as compared to the second criterion and controls both accuracy 
and convergence of the estimator. The great advantage of the proposed method is that a large 

25 first set of candidate vectors can be used, while the pre-selection method picks the most 

promising from that set for the actual evaluation of the similarity measure. The results require 
hardly more calculations than necessary for an identical state-of-the-art local vector estimator 
with a reduced first set of candidate vectors, but the chances of having promising candidates 
amongst the vectors for which the similarity measure is evaluated have significantly 

30 increased. 

A further advantage of the method becomes clear when considering a 
dedicated hardware implementation. Such an implementation often cannot profit from an 
operations count which is low on the average. It has to be designed for foe worst oase 
situation. Now, with a limited first set of candidate vectors, there is a good chance that the 



10. JAN. 2003 138 29 

PHNL03004SEPP 



10 



15 



20 



25 



30 



PS CIP NL +31 40 2743489 




NR. 980 P.ia<23 
10 10.01.2003 13:30:3 



4 09.01.2003 
number of actually different candidate vectors is lower than the capacity of the hardware. 
With a larger first set followed by a pre-selection module as proposed in this invention this 
chance can be much decreased. This leads to a more optimal use of the capacity of the 
hardware. 

Optionally the second set of candidate motion vectors is extended with 
candidate motion vectors which are not comprised by the firat set of motion vectors. E-g. the 
null-vector, i.e. no motion, is added or a candidate motion vector which is based on the 
median of the selected motion vectors of the first set of motion vectors. 

The candidate vectors in the first set of candidate vectors are preferably 
Spatially andVor temporally predicted based on already determined estimated local vectors 
and/or the zero vector and/or update vectors, which are either random vectors or belong to a 
limited fixed set of update vectors. Assuming that objects in a picture of an image sequence 
are larger than a block and have inertia, local vectors of a current block are quite likely to be 
similar to already detetmined local vectors in other neighboring blocks of the current picture 
around the current block (spatial predictions) or to already detaniined local vectors of 
neighboring blocks in die previous image (temporal predictions). The zero vector as 
candidate vector is particularly helpM tor picture parts without motion, whereas the addition 
of update vectors to spatially and/or temporally predicted local vectors solves the problem 
that in the initialization phase, all local vectors on which the prediction couldbe based aw 



According to the invention, the local vectors preferably represent motion 
vectors that describe the motion of groups of pixels inpictures of an image sequence: 

At least one of said motion vectors may be predicted according to a parametric 
2D global motion model. For instance, expressing* motion vector as 2D first-order equation, 
camera motion such as panning, tilting, travelling and zooming can be precisely modeled. 
This type of motion has a regular character, causing smooth motion vectors as compared to 
object motion. Whereas zooming generates motion vectors that linearly change with the 
spatial position, panning, tilting and travelling generate a uniform motion vector tor the entire 
picture. If such global motion occurs, it can be more efficient to estimate the parameters of 
the parametric 2D global motion model instead of the motion vectors themselves. 

The local vectors can also represent sets of parameters that describe the 
motion model of a group of pixels in pictures of an image sequence. 

As a further alternative, the local vectors may represent spatial features of a 
group of pixels, in particular texture, dynamic range, color, or average value. 
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According to the invention, the second criterion can he implemented as a 
match error criterion such as the Sum of Absolute Differences (SAD) criterion, or as the 
Mean Square Error (MSB) criterion. In the context of motion vector estimation, then the SAD 
or MSE between pixels or groups of pixels of the predicted and the current image is 
5 calculated. In contrast, in the context of image segmentation, where the components of the 
local vectors represent spatial features such as texture, dynamic range, color, or average 
luminance value of an image segment, the SAD and MSE criteria are directly applied to the 
components of the local vectors and the corresponding spatial features that are measured 
fiom the local image content 

10 The selection of candidate vectors fiom the first set of candidate vectors to 

torm a smaller second set of candidate vectors is suitably based on a ranking of the 
corresponding vector components of the candidate vectors in the first set of candidate 
vectors. 

The selection of candidate vectors fiom the first set of candidate vectors to 
15 term a smaller second set of candidate vectors can also be based on a ranking of the 
candidate vectors in the first set of candidate vectors. 

Jn a preferred embodiment of the invention, the second set of candidate 
vectors contains at least one extreme and/or one least extreme candidate vector of the first set 
of candidate vectors according to the first criterion. 

As the least extreme candidate vector is often a good one in the converged 
situation, while the more extreme vectors are particularly helpful in the un-converged 
station, it makes sense to select only these for evaluation with the subsequent, 
computationally more expensive, second criterion. Adding the zero vector (indicating no 
motion) as an extreme vector also makes sense, as the interpolation of stationary picture parts 
25 is critical In many applications of motion vectors, 

The extreme candidate vectors are preferably the two vectors with the largest 
distance to the average vector of a number of candidate vectors of the first set of candidate 
vectors or with the largest distance to a spatial prediction vector in the first set of candidate 
vectors, or the longest and the shortest vector, or the largest distance to the rest of the 
30 candidate vectors of the first set of candidate vectors. 

The least extreme candidate vector is preferably the vector with the smallest 
distance to the average vector of a number of candidate vectors of toe first set of candidate 
vectors or with the smallest distance to a spatial prediction vector in the first set of candidate 
vectors, or the vector median. 
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A farther preferred embodiment of the invention is a device for recursively 
estimating local vectors from at least one picture taken from an image sequence, consisting of 
means to generate a fbest set of candidate vectors under at least partial use of recursion, means 
to select candidate vectors from the first set of candidate vectors according to a first criterion 
to fbim a smaller second set of candidate vectors, means to evaluate the candidate vectors of 
the second set of candidate vectors tor a group of pixels based on a second criterion, means to 
determine the best vectors from the second set of candidate vectors according to said second 
criterion and means to assign said determined best vectors to a group of pixels that is related 
to the group of pixels the candidate vectois of the second set of candidate vectors were 
evaluated for. 

A last preferred embodiment of the present invention is a computer program 
product directly loadable into the internal memory of a digital computer, comprising software 
code portions for performing the stqas of generating a first set of candidate vectors under at 
least partial use of recursion, selecting candidate vectors from the first set of candidate 
vectors according to a first criterion to form a smaller second set of candidate vectors, 
evaluating the candidate vectors of the second set of candidate vectors for a group of pixels 
based on a second criterion, determining the best vectors from the second set of candidate 
vectors according to said second criterion and assigning said determined best vectois to a 
group of pixels that is related to the group of pixels the candidate vectors of the second set of 
candidate vectors were evaluated for, when said product is run on a computer. 



25 



These and other aspects of the invention will be apparent from and elucidated 
with reference to the embodiments described hereinafter. In the figures Show: 

Fig, 1 : a first embodiment of a recursive BMA according to flic invention, 
where motion vectois are estimated as local vectors, and 

Kg. 2: a second embodiment of a recursive BMA according to the invention, 
where the estimation of motion vectors as local vectors is enhanced by integrating a 
candidate motion vector that is predicted according to a global motion model 



30 



Kg. 1 shows arecursive BMA for the estimation of motion vector according 
to a first embodiment of the invention. 
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Let DfrO denote the field of motion vectors between, the current image 
and the previous image w -1) of an image sequence, where * = [x,y] T is the pixel grid 
vector. Further let 3(£ 9 ri) e 6(w) indicate the motion vector assigned to an AT x r block 
B(X) of pixels in the current image J(x 9 n) „ where the center of the block is identified by the 
5 blocfc grid vector X*\x„X y ] T . 

As shown in Fig. 1 , the prediction memory instance 1 outputs a set of 
candidate vectors 

where the candidate vectors C ~ [c, 4 C y ] r are limited to the discrete candidate set 

io as™ - {c|- jv <; c, <n>-ms c y sm} 7 

with constant pre-defined integers iV and M. 

Note that there exists a variety of different choices on the composition of the 

set of candidate vectors CS(X,n) and the updating procedure presented in the sequel of the 

description of this preferred embodiment This specific embodiment, which assumes that 
IS blocks in a picture are scanned ftom top left to bottom right and assumes temporal and/or 

spatial consistency, thus should only be regarded as an example for a much more general 

local vector estimation principle. 

Either of the 4 spatial candidate vectors of the presented set of candidate 

vectors CS(X,n) , ie. tho vectors that depend only on the index k or are independent of all 
20 indices h> is then fed to the update instance 2, whore an update vector XJ0t 9 n) is added. 

Adding an update vector to one of the spatial candidate vectors contained in CS(X: 9 n) solves 

the problem that in the initialization phase of the recuzsion, all vectors equal the 0 vector. 
Update vectors can either be generated as noise vectors, or, simpler, be taken from a limited 
fixed update set stored in a look-up-table, such as 

if pixel resolution with integer update values is desired, or such as 
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for quarterpixel resolution with fiacttoaat update values. 

Inftocont^ *„ =M**d * =f0»irdeno te tb e 2Dorfho n onnal basis 



vectors. 
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s(C,Z,n)~ E |/(^»)-J r (2-C,»-l)|+a|p(^»)J, 

where |j7(-f", »)| is the length of the update vector, a is a constant, and the matching error is 

summed overablock at position ^[ar^-Sr,] 3 ' ofthe block grid witii a widflxX and 
height F , defined as 

5 J5(l) -ty JT, -Xj2ZxZX x +Xl2> X v -Y/2£y£X y +7/2). 

Alternatively, a different similarity measure such as the Mean Square Error (MSB) can be 
applied as second criterion as well. Note that* instead of evaluating the similarity measure for 
all pixels x = [x,y] 7 on the pixel grid within the block B (X) , spatial sub-sampling in both 
x- and y-diieotion can be performed before evaluating the similarity measure to reduce the 
10 number of computations, where* of course, some accuracy is lost 

Irrespective ofthe applied second criterion and sampling technique, the best 
vector selection instance 7 further selects the candidate motion vector leading to the largest 
similarity measure: 

D(X,n) m {5 e CS^&n^siC&n) < e<? t X 9 n)\v {p a CS^&n)} 

IS and assigns this best candidate motion vector to all pixels at positions x = [x f y] T on the pixel 
grid within the block B(jt) (even if spatial sub-sampling was performed to reduce the 
computational effort in evaluating the similarity measure). 

The best motion vector 3(X>n) then is output as result of the motion 
estimation for block B(X) , but also stored in the prediction memory instance 7 for use in 

20 subsequent reoursion steps. 

Fig. 2 shows a second preferred embodiment of the present invention, where 
motion vectors are estimated as local vectors and where the recursive estimation is enhanced 
by integrating a candidate motion vector that is predicted according to a global motion 
modeL 

25 Basically, the set up of Fig, 2 evolves from the set-up of Fig. 1, in that the set- 

up of Fig, 2 comprises a prediction memory instance 1 , an update instance 2, an update 
generator instance 3» composed of amod-jp-count 4 and a look-up-table 5, a preselection 
instance 6 and a best vector selection instance 7. 

As in the first preferred embodiment ofthe invention shown in Fig. 1, a first 

30 set of candidate motion vectors CS(X,n) is spatially and temporally predicted by the 
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memory 1 for mo now recursion step. 
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memory only. 
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together with the spatial (some of which may he updated) and temporal predictions fiom the 
prediction memory instance 1. 
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1. Method for lively bating local vectors from at least one picture taken 

com an imago sequence, comprising the steps of 

- generating a first set of candidate vectors wider at least partial use of 

recursion, 

toarw^ "^^"^vec^ 
to a first criterion to form a smaller second set of candidate vectors, 

- evaluating the candidate vectors of the second set of candidate vectors tor a 
group of pixels based on a second criterion, 

- determining the best v^tors from the second set of candidate vectors 
10 according to said second criterion and 

evaluated for. 
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5, Method according to claim 1 or 2 ? characterized in that the local vectors 

represent spatial features of a group of pixels, in particular texture, dynamic range, color or 
average value. 



5 7, Method according to claim 1-6, characterized in that the second criterion is a 

match error criterion such as the Sum of Absolute Differences (SAD) criterion, or a mean 
square error criterion. 

g. Method according to claim 1-7, characterized in that the selection of candidate 

10 vectors from the first set of candidate vectors to form a smaller second set of candidate 
vectors is based on a ranking of the corresponding vector components of the candidate 
vectors in the first set of candidate vectors. 

9. Method according to claim 1-7, characterized in that the selection of candidate 
IS vectors from the first set of candidate vectors to form a smaller second set of candidate 

vectors is based on a ranking of the candidate vectors in the first set of candidate vectors. 

10. Method according to claim 1-9, characterized in that the second set of 
candidate vectors contains at least one extreme and/or one least extreme candidate vector of 

20 the first set of candidate vectors according to the first criterion. 

1 1. Method according to claim 10, characterized in thai the extreme candidate 
vectors are the two vectors with the largest distance to the average vector of a number of 
candidate vectors of the first set of candidate vectors or with the largest distance to a spatial 

25 prediction vector in the first set of candidate vectors, or the longest and the shortest vector, or 
the largest distance to the rest of the candidate vectors of the first set of candidate vectors. 

12. Method according to claim 10, characterized in that the least extreme 
candidate vector is the vector with the smallest distance to the average vector of a number of 

30 candidate vectors of the first set of candidate vectors or with the smallest distance to a spatial 
prediction vector in the first set of candidate vectors, or the vector median, 



13. Device for recursively estimating local vectors from at least one picture taken 

from an image sequence, consisting o£ 
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• means to generate a first set of candidate vectors nnder at least partial use of 



- means to select candidate vectors from the first set of candidate vectors 
according to a first criterion to form a smaller second set of candidate vectors, 

5 - means to evaluate the candidate vectors of the second set of candidate 

vectors for a group of pixels based on a second criterion, 

- means to determine the best vectors ftom me second set of candidate vectors 
according to said second criterion and 



: vectors 



10 * " mQ!ms to ~ Des * vectors to a group of pixels that is 

10 ^tothegmimof^ 

were evaluated for. 



14. 



digital computer, comprising software code portions 
15 said product is run on a computer. 



tbe internal memory of a 
s of claim 1 when 
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ABSTRACT: 



The invention relates to a method for recursively estimating local vectors from 
at least one picture taken from an image sequence. To reduce the computational complexity 
of the estimation method without deteriorating its accuracy, it is proposed that the method 
comprises the steps of generating a first set of candidate vectors under at least partial use of 

5 recursion, selecting candidate vectors from the first set of candidate vectors according to a 
first criterion to form a smaller second set of candidate vectors, evaluating the candidate 
vectors of the second set of candidate vectors for a group of pixels based on a second 
criterion, determining the best vectors from the second set of candidate vectors according to 
said second criterion and assigning said determined best vectors to a group of pixels that is 

10 related to the group of pixels the candidate vectors of the second set of candidate vectors 
were evaluated for. The invention ferfher relates to a device for recursively estimating local 
vectors from at least one picture taken from an image sequence, and to a computer program 
product comprising software code portions &r recursively estimating local vectors from at 
least one picture taken from an image sequence. 
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